Bugfix #2070 var list numeric order #2072

georgemccabe · 2023-03-02T20:24:52Z

Ensures that parse_var_list returns field info in numeric order (1,2,...,9,10,11) instead of alphabetical (1,10,11,2,3,etc)

Pull Request Testing

Describe testing already performed for these changes:

Added test to confirm behavior, then applied fix to code to pass test

Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:

Ensure all automated tests pass

Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]
Do these changes include sufficient testing updates? [Yes]
Will this PR result in changes to the test suite? [No]
Please complete this pull request review by 3/6/2023.

Pull Request Checklist

See the METplus Workflow for details.

Add any new Python packages to the METplus Components Python Requirements table.
Review the source issue metadata (required labels, projects, and milestone).
Complete the PR definition above.
Ensure the PR title matches the feature or bugfix branch name.
Define the PR metadata, as permissions allow.
Select: Reviewer(s)
Select: Organization level software support Project or Repository level development cycle Project
Select: Milestone as the version that will include these changes
After submitting the PR, select Development issue with the original issue number.
After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
Close the linked issue and delete your feature or bugfix branch from GitHub.

…phabetical order (VAR1, VAR10, VAR2, etc.) instead of numeric

…epresentation of the integer index

JohnHalleyGotway

I approve of these changes.

I tested this fix in seneca:/d1/projects/METplus/pull_requests/METplus-5.1.0/bugfix_2070
By running the medium_range/GridStat_fcstGFS_obsGFS_Sfc_MultiField.conf use case.

conda activate /home/met_test/.conda/envs/metplus_dev.v5.1

cd /d1/projects/METplus/pull_requests/METplus-5.1.0/bugfix_2070
mkdir -p OUTPUT/develop OUTPUT/bugfix

export METPLUS_TEST_MET_INSTALL_DIR=/usr/local/met-11.0.0
export METPLUS_TEST_INPUT_BASE=/d1/projects/METplus/METplus_Data

# Run current develop branch
export METPLUS_TEST_OUTPUT_BASE=`pwd`/OUTPUT/develop
METplus/ush/run_metplus.py  -c system.conf METplus/parm/use_cases/model_applications/medium_range/GridStat_fcstGFS_obsGFS_Sfc_MultiField.conf

# Run bugfix branch
export METPLUS_TEST_OUTPUT_BASE=`pwd`/OUTPUT/bugfix
METplus-bugfix/ush/run_metplus.py  -c system.conf METplus-bugfix/parm/use_cases/model_applications/medium_range/GridStat_fcstGFS_obsGFS_Sfc_MultiField.conf

Compared the order of the output variables written to the .stat file:

grep SL1L2  OUTPUT/develop/met_out/GFS/sfc/201706130000/grid_stat/grid_stat_GFS_TMP_vs_ANLYS_TMP_Z2_240000L_20170613_000000V.stat | grep CONUS | awk '{print $10, "/", $12}' | sed -r 's/$/, /g' | tr -d '\n'

Order of output variables for develop:

TMP / Z2, TSOIL / Z10-0, SOILW / Z10-0, WEASD / Z0, CAPE / Z0, CWAT / L0, PWAT / L0, TMP / L0, HGT / L0, TOZNE / L0, RH / Z2, SPFH / Z2, HPBL / L0, PRES / Z0, PRMSL / L0, TMP / Z0, UGRD / Z10, VGRD / Z10

Order of output variables for bugfix:

TMP / Z2, RH / Z2, SPFH / Z2, HPBL / L0, PRES / Z0, PRMSL / L0, TMP / Z0, UGRD / Z10, VGRD / Z10, TSOIL / Z10-0, SOILW / Z10-0, WEASD / Z0, CAPE / Z0, CWAT / L0, PWAT / L0, TMP / L0, HGT / L0, TOZNE / L0

I confirmed that the bugfix output order now matches the order in which the variables are listed in GridStat_fcstGFS_obsGFS_Sfc_MultiField.conf. So they are now being processed in numeric order rather than alphabetic order.

Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: cristianastan2 <cstan@gmu.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: bikegeek <minnawin@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> fixing some spacing #770 Fixing formatting, typos etc. #770 Fixing formatting #770 fixing typos, spacing, etc. #770 fixing typos, grammar and punctuation. #770 Fixing TOC and coding block #770 fixing note #770 fix. Updating bolding and italics for file names and directories #770 fixing space #770 fixing conflict #2 #770 fixing typos #770 fixing typos #2 #770 fixing formatting #770 fixing errors #650 fixing errors take 2 #650 fixing errors take 3 #650 fixing errors take 4 #650 fixing errors take 5 #650 fixing question order #650 fix #1706 fix PhaseDiagram use case to avoid writing into INPUT_BASE (#1708) fix #1713 develop METPLOTPY_BASE (#1715) fix #1691 remove whitespace from output file paths (#1721) fix Contributor's Guide GitHub Workflow page (#1774) fix release (#1790) fix GitHub Actions warnings (#1864) fix #1884 develop PCPCombine {custom} in subtract method (#1887) fix #1939 develop - failure reading obs when zipped file also exists (#1941) Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099)

Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: cristianastan2 <cstan@gmu.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: bikegeek <minnawin@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> fixing errors #650 fixing errors take 2 #650 fixing errors take 3 #650 fixing errors take 4 #650 fixing errors take 5 #650 fixing question order #650 fix #1706 fix PhaseDiagram use case to avoid writing into INPUT_BASE (#1708) fix #1713 develop METPLOTPY_BASE (#1715) fix #1691 remove whitespace from output file paths (#1721) fix Contributor's Guide GitHub Workflow page (#1774) fix release (#1790) fix GitHub Actions warnings (#1864) fix #1884 develop PCPCombine {custom} in subtract method (#1887) fix #1939 develop - failure reading obs when zipped file also exists (#1941) Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) Fix for Dockerfile smell DL4000 (#2112) fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) fix #2137 develop PointStat -obs_valid_beg/end (#2141)

Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: cristianastan2 <cstan@gmu.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: bikegeek <minnawin@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> fixing errors #650 fixing errors take 2 #650 fixing errors take 3 #650 fixing errors take 4 #650 fixing errors take 5 #650 fixing question order #650 fix #1706 fix PhaseDiagram use case to avoid writing into INPUT_BASE (#1708) fix #1713 develop METPLOTPY_BASE (#1715) fix #1691 remove whitespace from output file paths (#1721) fix Contributor's Guide GitHub Workflow page (#1774) fix release (#1790) fix GitHub Actions warnings (#1864) fix #1884 develop PCPCombine {custom} in subtract method (#1887) fix #1939 develop - failure reading obs when zipped file also exists (#1941) Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) Fix for Dockerfile smell DL4000 (#2112) fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) fix #2137 develop PointStat -obs_valid_beg/end (#2141) fix failured introduced by urllib3 (see urllib3/urllib3#2168) fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162)

Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: cristianastan2 <cstan@gmu.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: bikegeek <minnawin@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: mrinalbiswas <biswas@ucar.edu> fixing errors #650 fixing errors take 2 #650 fixing errors take 3 #650 fixing errors take 4 #650 fixing errors take 5 #650 fixing question order #650 fix #1706 fix PhaseDiagram use case to avoid writing into INPUT_BASE (#1708) fix #1713 develop METPLOTPY_BASE (#1715) fix #1691 remove whitespace from output file paths (#1721) fix Contributor's Guide GitHub Workflow page (#1774) fix release (#1790) fix GitHub Actions warnings (#1864) fix #1884 develop PCPCombine {custom} in subtract method (#1887) fix #1939 develop - failure reading obs when zipped file also exists (#1941) Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) Fix for Dockerfile smell DL4000 (#2112) fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) fix #2137 develop PointStat -obs_valid_beg/end (#2141) fix failured introduced by urllib3 (see urllib3/urllib3#2168) fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) fix #2168 develop - StatAnalysis time shift (#2169) fix releases. (#2183) fix #2189 develop - spaces in complex thresholds (#2191)

@georgemccabe

* fixed TCPairs unit tests to handle default value for match_points * bugfix #1691 remove whitespace from output file paths (#1721) * added variables that are needed for use case and were accidentally removed * log error and increment error count if required lat/lon bound variables are not set instad of calling sys.exit * skip metplus_final.conf in check for new output to prevent false alarm diffs * Update index.rst replace METdatadb with METdataio * Update metdatadb.rst Replace metdatadb with metdataio * renamed from metdatadb * replacing all other instances of metdatdb/METdatadb with metdataio/METdataio * after renaming directory from metdatadb to metdataio * replace METdatadb with METdataio * copied and replaced with metdataio filenames * replaced with metdataio version * Update the Release Guide while creating the MET-11.0.0-beta2 development release. Automatic Zenodo updates for each MET release have been DISABLED since they were being created for each development release. Revise the MET instructions to manually update the MET version on Zenodo only for official releases, not development or bugfix releases. * Changed version number * Added release notes * Update the labels scripts by adding 1 new one and renaming metdatadb to metdataio. * updated pandas version used for METdataio use cases to use 1.2.3 instead of 1.4.3 * update version to show development towards beta3 release * Changed METdatadb references to METdataio * Adding modulefiles for installation on various machines. (#1750) * Feature 1746 recreate release (#1751) * Adding instructions for recreating and existing release * Reorder the step's based on suggestions from @georgemccabe * Feature #1489 plot point obs wrapper (#1753) * Fixing typo * Feature #1657 clean up directory creation (#1756) * Feature update labels (#1765) * Modifications for METbaseimage * Update README.md Co-authored-by: johnhg <johnhg@ucar.edu> * feature dtcenter/METplus-Internal#20 add user ID to log output (#1771) * Per dtcenter/METplus-Internal#20, change log output to always include username and include user ID if it is available * Per dtcenter/METplus-Internal#20, fixed bug introduced from commit 458714c to develop where error is thrown if username cannot be obtained, which occurs in environment running GHA tests * Revert "Per dtcenter/METplus-Internal#20, fixed bug introduced from commit 458714c to develop where error is thrown if username cannot be obtained, which occurs in environment running GHA tests" This reverts commit fb2d093. * Revert "Per dtcenter/METplus-Internal#20, change log output to always include username and include user ID if it is available" This reverts commit 458714c. * per dtcenter/METplus-Internal#20, implemented working solution to user info logging * change file path based on change to MODE output from dtcenter/MET#2251 * fix Contributor's Guide GitHub Workflow page (#1774) * Feature dtcenter/METplus-Internal#32 add timestamp to final conf and file_lists (#1763) * Feature dtcenter/METplus-Internal#15 checksum for release (#1769) Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature dtcenter/METplus-Internal#32 change SCRUB_STAGING_DIR default and add documentation (#1784) * fix to allow wildcard characters in UserScript input templates, ci-run-all-diff * per #1787, allow NA instead of only integers for climo_mean/stdev hour_interval * per #1787, added test for setting NA for climo_mean hour_interval * Updating note format * update Python version to 3.8.6 * Revert "update Python version to 3.8.6" This reverts commit c1e6cbc. * added return value checking to error and exit a non-zero value if docker build or docker push fail * update instructions to find issues for a bugfix release (#1790) * Feature dtcenter/METplus-Internal#34 set the log time to UTC/GMT (#1793) * Feature #1330 MJO ENSO use case (#1794) Co-authored-by: cristianastan2 <cstan@gmu.edu> * Feature #1816 ensemble vx removal from EnsembleStat (#1820) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature #1816 rename cases and files (#1823) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Move files under internal_tests into internal/tests (#1818) * info for new use case * renamed picture of unrelated use case to properly display * Minor tweak to the release guide instructions for MET development releases. * Feature #1817 5.0.0-beta3 release (#1825) Co-authored-by: bikegeek <minnawin@ucar.edu> * Update VERSION * updated file name * updated MET file locations * explicitly set units for obs read via python embedding since PR dtcenter/MET#2288 no longer guesses units based on grib table units (which can be inaccurate of the actual data) * turn off precip use case group * Feature #1829 set unique ID in config (#1830) * fixed typo in use case setting that causes differences in the test output * Feature #1687 RuntimeFreq read only files needed for given run (#1841) * Feature 1847 release (#1850) * Per #1847, moved release notes to be its own chapter, reformatted component release notes, and added new section for upgrade instructions * Per #1847, added release-notes internal tag * Per #1847, fix error and reformat link * Per #1847, modified release instructions to add a section about upgrade instructions * Per #1847, modified release instructions to add a section about upgrade instructions * Per #1847, resolve doc warnings * Per #1847, updated instructions for clarity. * Per #1847, modified text and link. * Per #1847, modified wording and resolve error with note * Feature #1863 fix GitHub Actions warnings (#1864) * Feature #1874 prevent false diffs (#1875) * Feature 1842 StatAnalysis - multiple issues (#1869) * Bugfix #1884 develop PCPCombine {custom} in subtract method (#1887) * Feature 1852 use case physics tendency (#1881) * Issue #1852 initial work for the Planview plot * Issue #1852 planview plot for physics tendency * issue #1852 cleared away unused/unneccessary code * Issue #1852 update script command * Issue #1852 first draft, doesn't have correct content * Issue #1852 updated name of output plot to include short_range in name * Issue #1852 renamed to include fcstOnly label * Issue #1852 initial commit * Issue #1852 * Issue #1852 * Issue #1852 initial commit * issue #1852 vertical cross section * Issue 1852 clean up unused lines * Issue #1852 initial commit * Issue #1852 * Issue #1852 updates * Issue #1852 fixed documentation warnings, but some problems remain * fixing line length for chapter and section headers and one letter capitalization * cleaning up line spacing and trying to fix a critical error. * trying to fix critical error again * Issue #1852 more description for data and dependency on METplotpy with additional instructions * Issue #1852 update location of input data to single directory * Removing duplicate files with a misformatted name that are not needed * Per #1852, making changes to formatting to eliminate errors * Issue #1852 remove redundant yaml config file * Issue #1852 update PYTHONPATH information * Issue #1852 fixed conflicts * Issue #1852 update path to shapefile * Issue #1852 fixed copy and paste error * Issue #1852 add three use case entries: Planview, VerticalProfile, and VerticalCrossSection for the short range Physics tendency use case * Issue #1852 add latest short range tests to top, move down list when everything passes * Update use_case_groups.json move short_range use cases #10-12 to below the short range test for #9 * Issue #1852 clean up conf files by providing variables for the input netCDF files and for the planview plot, allow user to indicate pressure levels * Issue #1852 Indicate external Python packages * Issue #1852 added external dependency section * removed env vars that don't need to be set -- METPLOTPY_BASE is set by the automated test conf (.github/parm/test_settings.conf) and PYTHONPATH doesn't need to be set because METplotpy is installed in Python environment for use case tests * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_Planview.py Fixed one typo * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_VerticalCrossSection.py Fixed a typo * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_VerticalProfile.py Same typo, same fix * Issue #1852 remove extraneous entry for USER_SCRIPT_COMMAND, a hold-over from the original implementation which doesn't allow the user to easily specify the FV3 history file and grid spec netCDF file * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_VerticalProfile.conf Removed the incorrect USER_SCRIPT_COMMAND in previous commit. Reinstated the correct version and removed the incorrect command. * update html file from Planview to VerticalProfile Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> * Feature #1893 StatAnalysis custom looping (#1894) * Feature dtcenter/METplus-Internal#24 test failure (#1895) * updated version for creating release (#1897) * update release number for development towards beta5 * Adding one sentence to about daily follow up on discussions. * initial changes for new use case * updated paths, documentation * updated automated testing files * Feature #344 rearrange utils (#1900) * first additions for new use case, corrected documentation for old use case * Move Docker scripts to be consistent with other METplus components (#1907) * added picture * Correct spelling mistake * handle if input to format_thresh is already a list * Revert "handle if input to format_thresh is already a list" This reverts commit d329275. * handle if input to format_thresh is already a list * per #1910, added time util function to add field info to time info dictionary and updated wrapper logic to use this function to add this info instead of passing field info (var_info) to CommandBuilder.find_data to get level info, ci-run-all-diff * change logic to preserve full level string for [fcst/obs]_level so SeriesAnalysis cases build the same file names, but parse out time info for setting generic level for string sub to preserve previous behavior * try setting all input values in output dictionary, ci-run-all-diff * removed commented code * add quotes around input files to plot_point_obs command to allow support for Python Embedding as input * put all arguments on single line * remove commented code * per #1910, added unit tests to ensure that level is set properly in the forecast input template * added script to create geovista conda environment via Docker to use in the automated use case tests * Feature 1561 use case ugrid python embedding (#1927) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature 1453 use case iodav2 python embedding (#1930) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * fix permissions for script that were somehow changed to be not executable * Bugfix #1939 develop - failure reading obs when zipped file also exists (#1941) * Feature #1819 alert users if updates to EnsembleStat use cases are needed (#1945) Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * fixed underline that did not match title * Feature #1898 tc new options (#1932) close undefined * fixed broken doc util logic from bad PR merge * Feature #1569 EnsembleStat -ens_mean argument (#1952) * Feature #1953 seeps config in GridStat and PointStat (#1954) Co-authored-by: johnhg <johnhg@ucar.edu> * Feature #1566 update Python version to 3.8.6 (#1958) * Feature #1956 5.0.0-beta4 release (#1963) * Update VERSION * added warning in release notes that location of METplus Dockerfile has changed * combined release notes for all betas for the v5.0.0 release * updated tags for each METplus component to match the upcoming release * Feature #1885 gh_projects (#1973) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Switch from double to single quotes. * Update formatting of the GitHub project filters so that they can be copy/pasted directly from RTD into the GitHub project filter. * Feature #1953 PointStat output_flag.seeps and output_flag.seeps_mpr (#1976) * Feature #1711, #1783, #1814, and #1835 Documentation - Multiple files, PCPCombine constant init, Docker instructions, and MET links (#1967) Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * run diff logic on push for develop and main_v* branches, fix logic to properly run all use cases * fixed name of external * added Bugfix: to default title of new bugfix issue * added space after New Use Case: * changed references to METdatadb to METdataio that were missed * Update overview.rst Removed statsmodels from the table, this package is no longer a requirement. The necessary functionality was pulled out of the statsmodels repository and incorporated into the METcalcpy repository * Update overview.rst METdataio *must* only use pandas 1.2.3. METdataio also requires pyyaml. * Update overview.rst METdatio uses pytest to test aspects of the METreformat module * Update rotate_authorship.rst Explicitly add the step to commit and push changes to the file containing the author list. * Update overview.rst Added METdataio for lxml, numpy, pandas, and pymysql * Update finalize_release_on_github_official.rst added branch protection rules step and the interface to Github was modified, you need to click on the stacked '->' icon to switch default branches * CI: replace / with _ to prevent errors creating docker tags for tests * Update overview.rst included METdataio for python 3.8.6+ requirement * Update overview.rst psutil no longer needed in METplotpy and METcalcpy * Update overview.rst METcalcpy no longer requires imutils and allow versions 0.5.3 and above * Update overview.rst cleaned up formatting for imutils * change asterisk to double asterisk to match branches that include forward slash / * update version of certifi based on dependabot auto-generated PR #1982 * Feature 1819 format doc (#1981) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update use case keywords (#1984) * Feature #1980 Use the main_vX.Y of METplus components for main METplus tests (#1985) * Feature #1986 v5.0.0 release (#1987) Closes #1986 * updated version for development towards beta1 of 5.1.0 * allow rest of workflow to run even if job to update data volumes fails in case server is down * updated version for action to update input data volumes and added max number of DockerHub tag pages to search to 15 * updated release notes for official METplus release to untar the sample data for next version directory * Update release guide to create the input test data directory for official MET releases. * Follow the METplus naming conventions for updating the the DTC web server data. * install library needed for geovista package, remove editable flag so geovista code will be added directly to python instead of reading it from the source code location, ci-skip-all * Feature 685 log updates (#1992) * add script to create conda environment with MetPy 1.4 for SWPC use case * Bugfix develop Fix broken documentation links (#2004) * fixed bug to set environment variables in the RegridDataPlane environment instead of the PyEmbedIngest environment * Feature #2015 modulefiles (#2024) * Adding modulefile for WCOSS2 * Updated references of 4.1.1 to 5.0.0 * Feature #2005 updates to create v5.1 conda environments that use debian10 and Python 3.10.4 (#2043) * update versions to fix bugs for METplotpy/calcpy use cases * Bugfix #2026 develop StatAnalysis looping (#2028) * Feature #2022 update python version to 3.10.4 for repo files (#2047) * Feature #2022 Use Debian 10 / Python 3.10.4 in automated tests (#2050) * add use case group name to output path so that output and error logs will contain use case category to more easily see which case was run * remove -group0 from group names for output path * add manual workflow to update truth data * ci: set gh token on checkout to enable push, ci-skip-all * Set user name/email via git config to allow push * add token to checkout * Feature #2049 Multiple interp.type.method/width (#2051) * added missing argument to script if building envs locally * get main_vX.Y truth data if running on main_vX.Y branch * Updating README.md for label creation and adding bold to common_labels.txt * Updated color for bold label * Feature #2054 consensus.write_members in TCPairs (#2057) * Create LICENSE.md (#2061) * per #2006, fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) * Feature 2065 v5.1.0 beta1 (#2066) * updated version number * added release notes * Update docs/Users_Guide/release-notes.rst Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * updated version with -dev * adding sphinx design to 3 files so dropdown menus will work in the release notes. * fix version numbers for python requirements for building documentation * fixed tab vs spaces * Bugfix #2070 var list numeric order (#2072) * Adding files for jet and hera * added commented process list that was accidentally removed -- this is used if the pre-processing steps are needed * Bugfix #2087 develop docs_pdf (#2091) * Feature 2076 release dropdown menu (#2083) * test dropdowns * Update release-notes.rst Attempting to resolve WARNING * testing dropdowns * adding dropdown menus * testing dropdown remove from warning section * dropdown testing * dropdown menu tests * dropdown menus after the warnings section, the dropdown menu only has one space instead of the normal 2 spaces. If it has 2 spaces, it gets sucked into the warning section. * spaces with warning * no spaces for warning * Update release-notes.rst warning note has to be indented 1 space. Originally it was no spaces but then all of the dropdown menus end up in the warning message. If it's 2 spaces, as is normal, then the warning is indented, which is unnecessary. 1 space is slightly indented and keeps the dropdown menus separate. --------- Co-authored-by: jprestop <jpresto@ucar.edu> * Bugfix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) * Feature 1516 use case mvmode (#2094) Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Fix for Dockerfile smell DL4000 (#2112) Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> * Changed "PROJECT" to "CYCLE" ASSIGNMENT * Feature 2115 update use cases (#2133) * Bugfix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) * Bugfix #2137 develop PointStat -obs_valid_beg/end (#2141) * Feature #2143 5.1.0-beta2 release (#2144) * update release guide steps with new info based on changes to GitHub web interface * update version for development towards rc1 release * Feature #1977 ARGO use case (#2148) Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * Feature develop cycle change (#2150) * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * Added paths to ignore --------- Co-authored-by: jprestop <jpresto@ucar.edu> * Documentation: Update Truth Data instructions (#2154) * upgrade python to 3.10 for documentation builds to fix failured introduced by urllib3 (see urllib3/urllib3#2168) * update workflow to update input test data to use branch from pull down or enter branch by hand if branch does not exist in dtcenter/METplus repo, e.g. if updating data for a PR coming from a fork * Bugfix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) * changed order of workflow event triggering info to more easily see the repository and commit sha that triggered the workflow * fix docker compose install and call -- needed due to new Alpine Linux release 3.18.0 that does not include docker-compose in the apk repository * Bugfix #2168 develop - StatAnalysis time shift (#2169) * Feature 1978 update pyreq doc (#2177) * Updating for 5.1.0 * Adding greater than or equal to signs * Feature 2106 release support (#2182) * Added a paragraph for Release Support Policy * Added a sentence about contacting the team through the METplus GitHub Discussions Forum. * Added Release Support Policy info * Per #2159, modify instructions to reflect that we only want a coordinated release for the official releases and updates to the coordinated release section for the bugfix releases. (#2183) * Feature 2147 workflow doc (#2185) * Add new images for github workflow * Added PR section * Modified the Find the GitHub Issue to be Find or Create a GitHub issue * Trying out image keyword compared with figure keyword * Trying out various widths * Added new figures * Reverting to figures from images after modifying figures * Adding the old images which are better quality * Trying out new indention * Working on formatting again * Reducing image widths * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update github_workflow.rst --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * added scripts to create new conda environment for PANDA-C use cases * Bugfix #2189 develop - spaces in complex thresholds (#2191) * added release notes for completed issues for rc1 * Feature 2011 tci from cesm fluxnet2015 netcdf (#2192) Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Bugfix #2179 develop TCPairs fix -diag argument (#2187) * sort list of files to make command more readable and make logic more easily testable * fix unit test to expect alphabetical order of files * Feature #2197 MODE new multi-variate options (#2203) * added to release notes * added release notes for other items completed for rc1 release * Feature #1626 TCDiag Wrapper (#2201) * Feature #1610 first round of SonarQube fixes (#2200) * Feature #2195 create v5.1.0-rc1 release (#2204) * Update version for development towards official release * updated script to use metplotpy as base and install pygrib because conda was failing trying to install matplotlib in Docker for some strange reason * Feature 2188 use case panda c (#2202) * added first use case, need to finish docs and add to list * added documentation files, 3 new cases * added 5th use case, docs * final use case, corrected paths for output * Update GridStat_fcstGFS_obsERA5_lowAndTotalCloudFrac.py * removed chars, added space to last section --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: bikegeek <minnawin@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: cristianastan2 <cstan@gmu.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu>

Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: cristianastan2 <cstan@gmu.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: bikegeek <minnawin@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> fix Contributor's Guide GitHub Workflow page (#1774) fix release (#1790) fix GitHub Actions warnings (#1864) fix #1884 develop PCPCombine {custom} in subtract method (#1887) fix #1939 develop - failure reading obs when zipped file also exists (#1941) Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) Fix for Dockerfile smell DL4000 (#2112) fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) fix #2137 develop PointStat -obs_valid_beg/end (#2141) fix failured introduced by urllib3 (see urllib3/urllib3#2168) fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) fix #2168 develop - StatAnalysis time shift (#2169) fix releases. (#2183) fix #2189 develop - spaces in complex thresholds (#2191) fix #2179 develop TCPairs fix -diag argument (#2187) fixes (#2200)

Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: cristianastan2 <cstan@gmu.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: bikegeek <minnawin@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> fix Contributor's Guide GitHub Workflow page (#1774) fix release (#1790) fix GitHub Actions warnings (#1864) fix #1884 develop PCPCombine {custom} in subtract method (#1887) fix #1939 develop - failure reading obs when zipped file also exists (#1941) Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) Fix for Dockerfile smell DL4000 (#2112) fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) fix #2137 develop PointStat -obs_valid_beg/end (#2141) fix failured introduced by urllib3 (see urllib3/urllib3#2168) fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) fix #2168 develop - StatAnalysis time shift (#2169) fix releases. (#2183) fix #2189 develop - spaces in complex thresholds (#2191) fix #2179 develop TCPairs fix -diag argument (#2187) fixes (#2200) fix diff tests (#2217) fix automated tests (#2237) fix #2235 rename multivar_itensity to multivar_intensity_flag (#2236)

Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: cristianastan2 <cstan@gmu.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: bikegeek <minnawin@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> fix Contributor's Guide GitHub Workflow page (#1774) fix release (#1790) fix GitHub Actions warnings (#1864) fix #1884 develop PCPCombine {custom} in subtract method (#1887) fix #1939 develop - failure reading obs when zipped file also exists (#1941) Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) Fix for Dockerfile smell DL4000 (#2112) fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) fix #2137 develop PointStat -obs_valid_beg/end (#2141) fix failured introduced by urllib3 (see urllib3/urllib3#2168) fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) fix #2168 develop - StatAnalysis time shift (#2169) fix releases. (#2183) fix #2189 develop - spaces in complex thresholds (#2191) fix #2179 develop TCPairs fix -diag argument (#2187) fixes (#2200) fix diff tests (#2217) fix automated tests (#2237) fix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) fix #2241 Create directory containing -out_stat file (#2242) fix #2245 use unique run ID to name logger instance (#2247) fix #2244 develop fix diff tests (#2254)

Co-authored-by: cristianastan2 <cstan@gmu.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: bikegeek <minnawin@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: root <root@localhost> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> Co-authored-by: John Sharples <41682323+John-Sharples@users.noreply.github.com> fix release (#1790) fix GitHub Actions warnings (#1864) fix #1884 develop PCPCombine {custom} in subtract method (#1887) fix #1939 develop - failure reading obs when zipped file also exists (#1941) Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) Fix for Dockerfile smell DL4000 (#2112) fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) fix #2137 develop PointStat -obs_valid_beg/end (#2141) fix failured introduced by urllib3 (see urllib3/urllib3#2168) fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) fix #2168 develop - StatAnalysis time shift (#2169) fix releases. (#2183) fix #2189 develop - spaces in complex thresholds (#2191) fix #2179 develop TCPairs fix -diag argument (#2187) fixes (#2200) fix diff tests (#2217) fix automated tests (#2237) fix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) fix #2241 Create directory containing -out_stat file (#2242) fix #2245 use unique run ID to name logger instance (#2247) fix #2244 develop fix diff tests (#2254) fixture to set pytest tmpdir (#2261) fix #1853 develop - PointStat don't require mask variables to be set (#2262) fix #2279 develop - buoy station file from 2022 (#2280)

Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: root <root@localhost> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> Co-authored-by: John Sharples <41682323+John-Sharples@users.noreply.github.com> fix GitHub Actions warnings (#1864) fix #1884 develop PCPCombine {custom} in subtract method (#1887) fix #1939 develop - failure reading obs when zipped file also exists (#1941) Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) Fix for Dockerfile smell DL4000 (#2112) fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) fix #2137 develop PointStat -obs_valid_beg/end (#2141) fix failured introduced by urllib3 (see urllib3/urllib3#2168) fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) fix #2168 develop - StatAnalysis time shift (#2169) fix releases. (#2183) fix #2189 develop - spaces in complex thresholds (#2191) fix #2179 develop TCPairs fix -diag argument (#2187) fixes (#2200) fix diff tests (#2217) fix automated tests (#2237) fix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) fix #2241 Create directory containing -out_stat file (#2242) fix #2245 use unique run ID to name logger instance (#2247) fix #2244 develop fix diff tests (#2254) fixture to set pytest tmpdir (#2261) fix #1853 develop - PointStat don't require mask variables to be set (#2262) fix #2279 develop - buoy station file from 2022 (#2280)

Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: root <root@localhost> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> Co-authored-by: John Sharples <41682323+John-Sharples@users.noreply.github.com> fix GitHub Actions warnings (#1864) fix #1884 develop PCPCombine {custom} in subtract method (#1887) fix #1939 develop - failure reading obs when zipped file also exists (#1941) Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) Fix for Dockerfile smell DL4000 (#2112) fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) fix #2137 develop PointStat -obs_valid_beg/end (#2141) fix failured introduced by urllib3 (see urllib3/urllib3#2168) fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) fix #2168 develop - StatAnalysis time shift (#2169) fix releases. (#2183) fix #2189 develop - spaces in complex thresholds (#2191) fix #2179 develop TCPairs fix -diag argument (#2187) fixes (#2200) fix diff tests (#2217) fix automated tests (#2237) fix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) fix #2241 Create directory containing -out_stat file (#2242) fix #2245 use unique run ID to name logger instance (#2247) fix #2244 develop fix diff tests (#2254) fixture to set pytest tmpdir (#2261) fix #1853 develop - PointStat don't require mask variables to be set (#2262) fix #2279 develop - buoy station file from 2022 (#2280) fix (#2313)

Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: root <root@localhost> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> Co-authored-by: John Sharples <41682323+John-Sharples@users.noreply.github.com> fix #1884 develop PCPCombine {custom} in subtract method (#1887) fix #1939 develop - failure reading obs when zipped file also exists (#1941) Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) Fix for Dockerfile smell DL4000 (#2112) fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) fix #2137 develop PointStat -obs_valid_beg/end (#2141) fix failured introduced by urllib3 (see urllib3/urllib3#2168) fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) fix #2168 develop - StatAnalysis time shift (#2169) fix releases. (#2183) fix #2189 develop - spaces in complex thresholds (#2191) fix #2179 develop TCPairs fix -diag argument (#2187) fixes (#2200) fix diff tests (#2217) fix automated tests (#2237) fix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) fix #2241 Create directory containing -out_stat file (#2242) fix #2245 use unique run ID to name logger instance (#2247) fix #2244 develop fix diff tests (#2254) fixture to set pytest tmpdir (#2261) fix #1853 develop - PointStat don't require mask variables to be set (#2262) fix #2279 develop - buoy station file from 2022 (#2280) fix (#2313)

* Feature dtcenter/METplus-Internal#24 test failure (#1895) * updated version for creating release (#1897) * update release number for development towards beta5 * Adding one sentence to about daily follow up on discussions. * initial changes for new use case * updated paths, documentation * updated automated testing files * Feature #344 rearrange utils (#1900) * first additions for new use case, corrected documentation for old use case * Move Docker scripts to be consistent with other METplus components (#1907) * added picture * Correct spelling mistake * handle if input to format_thresh is already a list * Revert "handle if input to format_thresh is already a list" This reverts commit d329275. * handle if input to format_thresh is already a list * per #1910, added time util function to add field info to time info dictionary and updated wrapper logic to use this function to add this info instead of passing field info (var_info) to CommandBuilder.find_data to get level info, ci-run-all-diff * change logic to preserve full level string for [fcst/obs]_level so SeriesAnalysis cases build the same file names, but parse out time info for setting generic level for string sub to preserve previous behavior * try setting all input values in output dictionary, ci-run-all-diff * removed commented code * add quotes around input files to plot_point_obs command to allow support for Python Embedding as input * put all arguments on single line * remove commented code * per #1910, added unit tests to ensure that level is set properly in the forecast input template * added script to create geovista conda environment via Docker to use in the automated use case tests * Feature 1561 use case ugrid python embedding (#1927) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature 1453 use case iodav2 python embedding (#1930) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * fix permissions for script that were somehow changed to be not executable * Bugfix #1939 develop - failure reading obs when zipped file also exists (#1941) * Feature #1819 alert users if updates to EnsembleStat use cases are needed (#1945) Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * fixed underline that did not match title * Feature #1898 tc new options (#1932) close undefined * fixed broken doc util logic from bad PR merge * Feature #1569 EnsembleStat -ens_mean argument (#1952) * Feature #1953 seeps config in GridStat and PointStat (#1954) Co-authored-by: johnhg <johnhg@ucar.edu> * Feature #1566 update Python version to 3.8.6 (#1958) * Feature #1956 5.0.0-beta4 release (#1963) * Update VERSION * added warning in release notes that location of METplus Dockerfile has changed * combined release notes for all betas for the v5.0.0 release * updated tags for each METplus component to match the upcoming release * Feature #1885 gh_projects (#1973) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Switch from double to single quotes. * Update formatting of the GitHub project filters so that they can be copy/pasted directly from RTD into the GitHub project filter. * Feature #1953 PointStat output_flag.seeps and output_flag.seeps_mpr (#1976) * Feature #1711, #1783, #1814, and #1835 Documentation - Multiple files, PCPCombine constant init, Docker instructions, and MET links (#1967) Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * run diff logic on push for develop and main_v* branches, fix logic to properly run all use cases * fixed name of external * added Bugfix: to default title of new bugfix issue * added space after New Use Case: * changed references to METdatadb to METdataio that were missed * Update overview.rst Removed statsmodels from the table, this package is no longer a requirement. The necessary functionality was pulled out of the statsmodels repository and incorporated into the METcalcpy repository * Update overview.rst METdataio *must* only use pandas 1.2.3. METdataio also requires pyyaml. * Update overview.rst METdatio uses pytest to test aspects of the METreformat module * Update rotate_authorship.rst Explicitly add the step to commit and push changes to the file containing the author list. * Update overview.rst Added METdataio for lxml, numpy, pandas, and pymysql * Update finalize_release_on_github_official.rst added branch protection rules step and the interface to Github was modified, you need to click on the stacked '->' icon to switch default branches * CI: replace / with _ to prevent errors creating docker tags for tests * Update overview.rst included METdataio for python 3.8.6+ requirement * Update overview.rst psutil no longer needed in METplotpy and METcalcpy * Update overview.rst METcalcpy no longer requires imutils and allow versions 0.5.3 and above * Update overview.rst cleaned up formatting for imutils * change asterisk to double asterisk to match branches that include forward slash / * update version of certifi based on dependabot auto-generated PR #1982 * Feature 1819 format doc (#1981) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update use case keywords (#1984) * Feature #1980 Use the main_vX.Y of METplus components for main METplus tests (#1985) * Feature #1986 v5.0.0 release (#1987) Closes #1986 * updated version for development towards beta1 of 5.1.0 * allow rest of workflow to run even if job to update data volumes fails in case server is down * updated version for action to update input data volumes and added max number of DockerHub tag pages to search to 15 * updated release notes for official METplus release to untar the sample data for next version directory * Update release guide to create the input test data directory for official MET releases. * Follow the METplus naming conventions for updating the the DTC web server data. * install library needed for geovista package, remove editable flag so geovista code will be added directly to python instead of reading it from the source code location, ci-skip-all * Feature 685 log updates (#1992) * add script to create conda environment with MetPy 1.4 for SWPC use case * Bugfix develop Fix broken documentation links (#2004) * fixed bug to set environment variables in the RegridDataPlane environment instead of the PyEmbedIngest environment * Feature #2015 modulefiles (#2024) * Adding modulefile for WCOSS2 * Updated references of 4.1.1 to 5.0.0 * Feature #2005 updates to create v5.1 conda environments that use debian10 and Python 3.10.4 (#2043) * update versions to fix bugs for METplotpy/calcpy use cases * Bugfix #2026 develop StatAnalysis looping (#2028) * Feature #2022 update python version to 3.10.4 for repo files (#2047) * Feature #2022 Use Debian 10 / Python 3.10.4 in automated tests (#2050) * add use case group name to output path so that output and error logs will contain use case category to more easily see which case was run * remove -group0 from group names for output path * add manual workflow to update truth data * ci: set gh token on checkout to enable push, ci-skip-all * Set user name/email via git config to allow push * add token to checkout * Feature #2049 Multiple interp.type.method/width (#2051) * added missing argument to script if building envs locally * get main_vX.Y truth data if running on main_vX.Y branch * Updating README.md for label creation and adding bold to common_labels.txt * Updated color for bold label * Feature #2054 consensus.write_members in TCPairs (#2057) * Create LICENSE.md (#2061) * per #2006, fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) * Feature 2065 v5.1.0 beta1 (#2066) * updated version number * added release notes * Update docs/Users_Guide/release-notes.rst Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * updated version with -dev * adding sphinx design to 3 files so dropdown menus will work in the release notes. * fix version numbers for python requirements for building documentation * fixed tab vs spaces * Bugfix #2070 var list numeric order (#2072) * Adding files for jet and hera * added commented process list that was accidentally removed -- this is used if the pre-processing steps are needed * Bugfix #2087 develop docs_pdf (#2091) * Feature 2076 release dropdown menu (#2083) * test dropdowns * Update release-notes.rst Attempting to resolve WARNING * testing dropdowns * adding dropdown menus * testing dropdown remove from warning section * dropdown testing * dropdown menu tests * dropdown menus after the warnings section, the dropdown menu only has one space instead of the normal 2 spaces. If it has 2 spaces, it gets sucked into the warning section. * spaces with warning * no spaces for warning * Update release-notes.rst warning note has to be indented 1 space. Originally it was no spaces but then all of the dropdown menus end up in the warning message. If it's 2 spaces, as is normal, then the warning is indented, which is unnecessary. 1 space is slightly indented and keeps the dropdown menus separate. --------- Co-authored-by: jprestop <jpresto@ucar.edu> * Bugfix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) * Feature 1516 use case mvmode (#2094) Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Fix for Dockerfile smell DL4000 (#2112) Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> * Changed "PROJECT" to "CYCLE" ASSIGNMENT * Feature 2115 update use cases (#2133) * Bugfix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) * Bugfix #2137 develop PointStat -obs_valid_beg/end (#2141) * Feature #2143 5.1.0-beta2 release (#2144) * update release guide steps with new info based on changes to GitHub web interface * update version for development towards rc1 release * Feature #1977 ARGO use case (#2148) Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * Feature develop cycle change (#2150) * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * Added paths to ignore --------- Co-authored-by: jprestop <jpresto@ucar.edu> * Documentation: Update Truth Data instructions (#2154) * upgrade python to 3.10 for documentation builds to fix failured introduced by urllib3 (see urllib3/urllib3#2168) * update workflow to update input test data to use branch from pull down or enter branch by hand if branch does not exist in dtcenter/METplus repo, e.g. if updating data for a PR coming from a fork * Bugfix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) * changed order of workflow event triggering info to more easily see the repository and commit sha that triggered the workflow * fix docker compose install and call -- needed due to new Alpine Linux release 3.18.0 that does not include docker-compose in the apk repository * Bugfix #2168 develop - StatAnalysis time shift (#2169) * Feature 1978 update pyreq doc (#2177) * Updating for 5.1.0 * Adding greater than or equal to signs * Feature 2106 release support (#2182) * Added a paragraph for Release Support Policy * Added a sentence about contacting the team through the METplus GitHub Discussions Forum. * Added Release Support Policy info * Per #2159, modify instructions to reflect that we only want a coordinated release for the official releases and updates to the coordinated release section for the bugfix releases. (#2183) * Feature 2147 workflow doc (#2185) * Add new images for github workflow * Added PR section * Modified the Find the GitHub Issue to be Find or Create a GitHub issue * Trying out image keyword compared with figure keyword * Trying out various widths * Added new figures * Reverting to figures from images after modifying figures * Adding the old images which are better quality * Trying out new indention * Working on formatting again * Reducing image widths * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update github_workflow.rst --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * added scripts to create new conda environment for PANDA-C use cases * Bugfix #2189 develop - spaces in complex thresholds (#2191) * added release notes for completed issues for rc1 * Feature 2011 tci from cesm fluxnet2015 netcdf (#2192) Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Bugfix #2179 develop TCPairs fix -diag argument (#2187) * sort list of files to make command more readable and make logic more easily testable * fix unit test to expect alphabetical order of files * Feature #2197 MODE new multi-variate options (#2203) * added to release notes * added release notes for other items completed for rc1 release * Feature #1626 TCDiag Wrapper (#2201) * Feature #1610 first round of SonarQube fixes (#2200) * Feature #2195 create v5.1.0-rc1 release (#2204) * Update version for development towards official release * updated script to use metplotpy as base and install pygrib because conda was failing trying to install matplotlib in Docker for some strange reason * Feature 2188 use case panda c (#2202) * added first use case, need to finish docs and add to list * added documentation files, 3 new cases * added 5th use case, docs * final use case, corrected paths for output * Update GridStat_fcstGFS_obsERA5_lowAndTotalCloudFrac.py * removed chars, added space to last section * turn off 5/6 new air quality and comp use cases because they are causing the disk to fill up in the automated tests and only 1 is needed to test the new behavior. Reordered use case groups to be alphabetical * disabled other new air quality use case because the disk is still filling up from this case * Feature 2136 cross spectra (#2208) * per #2212, continue instead of return so other forecast leads will not be skipped * Feature #2210 MODE multi-variate variable rename (#2211) * Bugfix diff tests (#2217) * added change from NCEP produtil that was somehow not included in the changes pulled from their repo before starting SonarQube work * Feature #2228 v5.1.0-rc2 release (#2229) * update version after rc2 release to note development towards official 5.1.0 release * Feature #2188 move PANDA-C cases to clouds category and fix automated tests (#2237) Co-authored-by: j-opatz <jopatz@ucar.edu> * Feature 2198 use case amdar pbl (#2206) Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Bugfix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) * Feature #2230 MvMODE usecase doc updates (#2239) This updates the use-case documentation to go along with the multivariate MODE enhancements. * combine release notes from development releases for 5.1.0 release * fix indentation to prevent incorrect nesting of drop down menus * Bugfix #2241 Create directory containing -out_stat file (#2242) * Enhance diff logic to control decimal precision to prevent failures (#2243) * Bugfix #2245 use unique run ID to name logger instance (#2247) * per #2245, add METplusConfig class function that is called when object is deleted to close log handlers. This prevents OSError: [Errno 24] Too many open files from running all pytests * Feature #1626 TCDiag updates to wrapper and basic use case (#2248) Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> * move new use case into another group * Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root <root@localhost> * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: John Sharples <41682323+John-Sharples@users.noreply.github.com> Co-authored-by: root <root@localhost>

Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: root <root@localhost> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Tracy <tracy.hertneky@noaa.gov> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> Co-authored-by: John Sharples <41682323+John-Sharples@users.noreply.github.com> Co-authored-by: reza-armuei <144857501+reza-armuei@users.noreply.github.com> fix #1939 develop - failure reading obs when zipped file also exists (#1941) Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) Fix for Dockerfile smell DL4000 (#2112) fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) fix #2137 develop PointStat -obs_valid_beg/end (#2141) fix failured introduced by urllib3 (see urllib3/urllib3#2168) fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) fix #2168 develop - StatAnalysis time shift (#2169) fix releases. (#2183) fix #2189 develop - spaces in complex thresholds (#2191) fix #2179 develop TCPairs fix -diag argument (#2187) fixes (#2200) fix diff tests (#2217) fix automated tests (#2237) fix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) fix #2241 Create directory containing -out_stat file (#2242) fix #2245 use unique run ID to name logger instance (#2247) fix #2244 develop fix diff tests (#2254) fixture to set pytest tmpdir (#2261) fix #1853 develop - PointStat don't require mask variables to be set (#2262) fix #2279 develop - buoy station file from 2022 (#2280) fix (#2313) fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary

Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: root <root@localhost> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Tracy <tracy.hertneky@noaa.gov> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> Co-authored-by: John Sharples <41682323+John-Sharples@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: reza-armuei <144857501+reza-armuei@users.noreply.github.com> Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) Fix for Dockerfile smell DL4000 (#2112) fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) fix #2137 develop PointStat -obs_valid_beg/end (#2141) fix failured introduced by urllib3 (see urllib3/urllib3#2168) fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) fix #2168 develop - StatAnalysis time shift (#2169) fix releases. (#2183) fix #2189 develop - spaces in complex thresholds (#2191) fix #2179 develop TCPairs fix -diag argument (#2187) fixes (#2200) fix diff tests (#2217) fix automated tests (#2237) fix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) fix #2241 Create directory containing -out_stat file (#2242) fix #2245 use unique run ID to name logger instance (#2247) fix #2244 develop fix diff tests (#2254) fixture to set pytest tmpdir (#2261) fix #1853 develop - PointStat don't require mask variables to be set (#2262) fix #2279 develop - buoy station file from 2022 (#2280) fix (#2313) fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary fix bug described in pangeo-data/xESMF#246

Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: root <root@localhost> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Tracy <tracy.hertneky@noaa.gov> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> Co-authored-by: John Sharples <41682323+John-Sharples@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: reza-armuei <144857501+reza-armuei@users.noreply.github.com> Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) Fix for Dockerfile smell DL4000 (#2112) fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) fix #2137 develop PointStat -obs_valid_beg/end (#2141) fix failured introduced by urllib3 (see urllib3/urllib3#2168) fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) fix #2168 develop - StatAnalysis time shift (#2169) fix releases. (#2183) fix #2189 develop - spaces in complex thresholds (#2191) fix #2179 develop TCPairs fix -diag argument (#2187) fixes (#2200) fix diff tests (#2217) fix automated tests (#2237) fix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) fix #2241 Create directory containing -out_stat file (#2242) fix #2245 use unique run ID to name logger instance (#2247) fix #2244 develop fix diff tests (#2254) fixture to set pytest tmpdir (#2261) fix #1853 develop - PointStat don't require mask variables to be set (#2262) fix #2279 develop - buoy station file from 2022 (#2280) fix (#2313) fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary fix bug described in pangeo-data/xESMF#246

Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: root <root@localhost> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Tracy <tracy.hertneky@noaa.gov> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> Co-authored-by: John Sharples <41682323+John-Sharples@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: reza-armuei <144857501+reza-armuei@users.noreply.github.com> Co-authored-by: Mallory Row <mallory.row@noaa.gov> Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) Fix for Dockerfile smell DL4000 (#2112) fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) fix #2137 develop PointStat -obs_valid_beg/end (#2141) fix failured introduced by urllib3 (see urllib3/urllib3#2168) fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) fix #2168 develop - StatAnalysis time shift (#2169) fix releases. (#2183) fix #2189 develop - spaces in complex thresholds (#2191) fix #2179 develop TCPairs fix -diag argument (#2187) fixes (#2200) fix diff tests (#2217) fix automated tests (#2237) fix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) fix #2241 Create directory containing -out_stat file (#2242) fix #2245 use unique run ID to name logger instance (#2247) fix #2244 develop fix diff tests (#2254) fixture to set pytest tmpdir (#2261) fix #1853 develop - PointStat don't require mask variables to be set (#2262) fix #2279 develop - buoy station file from 2022 (#2280) fix (#2313) fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary fix bug described in pangeo-data/xESMF#246 fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls

* Update release guide to create the input test data directory for official MET releases. * Follow the METplus naming conventions for updating the the DTC web server data. * install library needed for geovista package, remove editable flag so geovista code will be added directly to python instead of reading it from the source code location, ci-skip-all * Feature 685 log updates (#1992) * add script to create conda environment with MetPy 1.4 for SWPC use case * Bugfix develop Fix broken documentation links (#2004) * fixed bug to set environment variables in the RegridDataPlane environment instead of the PyEmbedIngest environment * Feature #2015 modulefiles (#2024) * Adding modulefile for WCOSS2 * Updated references of 4.1.1 to 5.0.0 * Feature #2005 updates to create v5.1 conda environments that use debian10 and Python 3.10.4 (#2043) * update versions to fix bugs for METplotpy/calcpy use cases * Bugfix #2026 develop StatAnalysis looping (#2028) * Feature #2022 update python version to 3.10.4 for repo files (#2047) * Feature #2022 Use Debian 10 / Python 3.10.4 in automated tests (#2050) * add use case group name to output path so that output and error logs will contain use case category to more easily see which case was run * remove -group0 from group names for output path * add manual workflow to update truth data * ci: set gh token on checkout to enable push, ci-skip-all * Set user name/email via git config to allow push * add token to checkout * Feature #2049 Multiple interp.type.method/width (#2051) * added missing argument to script if building envs locally * get main_vX.Y truth data if running on main_vX.Y branch * Updating README.md for label creation and adding bold to common_labels.txt * Updated color for bold label * Feature #2054 consensus.write_members in TCPairs (#2057) * Create LICENSE.md (#2061) * per #2006, fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) * Feature 2065 v5.1.0 beta1 (#2066) * updated version number * added release notes * Update docs/Users_Guide/release-notes.rst Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * updated version with -dev * adding sphinx design to 3 files so dropdown menus will work in the release notes. * fix version numbers for python requirements for building documentation * fixed tab vs spaces * Bugfix #2070 var list numeric order (#2072) * Adding files for jet and hera * added commented process list that was accidentally removed -- this is used if the pre-processing steps are needed * Bugfix #2087 develop docs_pdf (#2091) * Feature 2076 release dropdown menu (#2083) * test dropdowns * Update release-notes.rst Attempting to resolve WARNING * testing dropdowns * adding dropdown menus * testing dropdown remove from warning section * dropdown testing * dropdown menu tests * dropdown menus after the warnings section, the dropdown menu only has one space instead of the normal 2 spaces. If it has 2 spaces, it gets sucked into the warning section. * spaces with warning * no spaces for warning * Update release-notes.rst warning note has to be indented 1 space. Originally it was no spaces but then all of the dropdown menus end up in the warning message. If it's 2 spaces, as is normal, then the warning is indented, which is unnecessary. 1 space is slightly indented and keeps the dropdown menus separate. --------- Co-authored-by: jprestop <jpresto@ucar.edu> * Bugfix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) * Feature 1516 use case mvmode (#2094) Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Fix for Dockerfile smell DL4000 (#2112) Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> * Changed "PROJECT" to "CYCLE" ASSIGNMENT * Feature 2115 update use cases (#2133) * Bugfix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) * Bugfix #2137 develop PointStat -obs_valid_beg/end (#2141) * Feature #2143 5.1.0-beta2 release (#2144) * update release guide steps with new info based on changes to GitHub web interface * update version for development towards rc1 release * Feature #1977 ARGO use case (#2148) Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * Feature develop cycle change (#2150) * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * Added paths to ignore --------- Co-authored-by: jprestop <jpresto@ucar.edu> * Documentation: Update Truth Data instructions (#2154) * upgrade python to 3.10 for documentation builds to fix failured introduced by urllib3 (see urllib3/urllib3#2168) * update workflow to update input test data to use branch from pull down or enter branch by hand if branch does not exist in dtcenter/METplus repo, e.g. if updating data for a PR coming from a fork * Bugfix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) * changed order of workflow event triggering info to more easily see the repository and commit sha that triggered the workflow * fix docker compose install and call -- needed due to new Alpine Linux release 3.18.0 that does not include docker-compose in the apk repository * Bugfix #2168 develop - StatAnalysis time shift (#2169) * Feature 1978 update pyreq doc (#2177) * Updating for 5.1.0 * Adding greater than or equal to signs * Feature 2106 release support (#2182) * Added a paragraph for Release Support Policy * Added a sentence about contacting the team through the METplus GitHub Discussions Forum. * Added Release Support Policy info * Per #2159, modify instructions to reflect that we only want a coordinated release for the official releases and updates to the coordinated release section for the bugfix releases. (#2183) * Feature 2147 workflow doc (#2185) * Add new images for github workflow * Added PR section * Modified the Find the GitHub Issue to be Find or Create a GitHub issue * Trying out image keyword compared with figure keyword * Trying out various widths * Added new figures * Reverting to figures from images after modifying figures * Adding the old images which are better quality * Trying out new indention * Working on formatting again * Reducing image widths * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update github_workflow.rst --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * added scripts to create new conda environment for PANDA-C use cases * Bugfix #2189 develop - spaces in complex thresholds (#2191) * added release notes for completed issues for rc1 * Feature 2011 tci from cesm fluxnet2015 netcdf (#2192) Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Bugfix #2179 develop TCPairs fix -diag argument (#2187) * sort list of files to make command more readable and make logic more easily testable * fix unit test to expect alphabetical order of files * Feature #2197 MODE new multi-variate options (#2203) * added to release notes * added release notes for other items completed for rc1 release * Feature #1626 TCDiag Wrapper (#2201) * Feature #1610 first round of SonarQube fixes (#2200) * Feature #2195 create v5.1.0-rc1 release (#2204) * Update version for development towards official release * updated script to use metplotpy as base and install pygrib because conda was failing trying to install matplotlib in Docker for some strange reason * Feature 2188 use case panda c (#2202) * added first use case, need to finish docs and add to list * added documentation files, 3 new cases * added 5th use case, docs * final use case, corrected paths for output * Update GridStat_fcstGFS_obsERA5_lowAndTotalCloudFrac.py * removed chars, added space to last section * turn off 5/6 new air quality and comp use cases because they are causing the disk to fill up in the automated tests and only 1 is needed to test the new behavior. Reordered use case groups to be alphabetical * disabled other new air quality use case because the disk is still filling up from this case * Feature 2136 cross spectra (#2208) * per #2212, continue instead of return so other forecast leads will not be skipped * Feature #2210 MODE multi-variate variable rename (#2211) * Bugfix diff tests (#2217) * added change from NCEP produtil that was somehow not included in the changes pulled from their repo before starting SonarQube work * Feature #2228 v5.1.0-rc2 release (#2229) * update version after rc2 release to note development towards official 5.1.0 release * Feature #2188 move PANDA-C cases to clouds category and fix automated tests (#2237) Co-authored-by: j-opatz <jopatz@ucar.edu> * Feature 2198 use case amdar pbl (#2206) Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Bugfix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) * Feature #2230 MvMODE usecase doc updates (#2239) This updates the use-case documentation to go along with the multivariate MODE enhancements. * combine release notes from development releases for 5.1.0 release * fix indentation to prevent incorrect nesting of drop down menus * Bugfix #2241 Create directory containing -out_stat file (#2242) * Enhance diff logic to control decimal precision to prevent failures (#2243) * Bugfix #2245 use unique run ID to name logger instance (#2247) * per #2245, add METplusConfig class function that is called when object is deleted to close log handlers. This prevents OSError: [Errno 24] Too many open files from running all pytests * Feature #1626 TCDiag updates to wrapper and basic use case (#2248) Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> * move new use case into another group * Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root <root@localhost> * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <tracy.hertneky@noaa.gov> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: John Sharples <41682323+John-Sharples@users.noreply.github.com> Co-authored-by: root <root@localhost> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: reza-armuei <144857501+reza-armuei@users.noreply.github.com> Co-authored-by: Tracy <tracy.hertneky@noaa.gov> Co-authored-by: Mallory Row <mallory.row@noaa.gov> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com>

@JohnHalleyGotway

* update versions to fix bugs for METplotpy/calcpy use cases * Bugfix #2026 develop StatAnalysis looping (#2028) * Feature #2022 update python version to 3.10.4 for repo files (#2047) * Feature #2022 Use Debian 10 / Python 3.10.4 in automated tests (#2050) * add use case group name to output path so that output and error logs will contain use case category to more easily see which case was run * remove -group0 from group names for output path * add manual workflow to update truth data * ci: set gh token on checkout to enable push, ci-skip-all * Set user name/email via git config to allow push * add token to checkout * Feature #2049 Multiple interp.type.method/width (#2051) * added missing argument to script if building envs locally * get main_vX.Y truth data if running on main_vX.Y branch * Updating README.md for label creation and adding bold to common_labels.txt * Updated color for bold label * Feature #2054 consensus.write_members in TCPairs (#2057) * Create LICENSE.md (#2061) * per #2006, fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) * Feature 2065 v5.1.0 beta1 (#2066) * updated version number * added release notes * Update docs/Users_Guide/release-notes.rst Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * updated version with -dev * adding sphinx design to 3 files so dropdown menus will work in the release notes. * fix version numbers for python requirements for building documentation * fixed tab vs spaces * Bugfix #2070 var list numeric order (#2072) * Adding files for jet and hera * added commented process list that was accidentally removed -- this is used if the pre-processing steps are needed * Bugfix #2087 develop docs_pdf (#2091) * Feature 2076 release dropdown menu (#2083) * test dropdowns * Update release-notes.rst Attempting to resolve WARNING * testing dropdowns * adding dropdown menus * testing dropdown remove from warning section * dropdown testing * dropdown menu tests * dropdown menus after the warnings section, the dropdown menu only has one space instead of the normal 2 spaces. If it has 2 spaces, it gets sucked into the warning section. * spaces with warning * no spaces for warning * Update release-notes.rst warning note has to be indented 1 space. Originally it was no spaces but then all of the dropdown menus end up in the warning message. If it's 2 spaces, as is normal, then the warning is indented, which is unnecessary. 1 space is slightly indented and keeps the dropdown menus separate. --------- Co-authored-by: jprestop <jpresto@ucar.edu> * Bugfix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) * Feature 1516 use case mvmode (#2094) Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Fix for Dockerfile smell DL4000 (#2112) Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> * Changed "PROJECT" to "CYCLE" ASSIGNMENT * Feature 2115 update use cases (#2133) * Bugfix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) * Bugfix #2137 develop PointStat -obs_valid_beg/end (#2141) * Feature #2143 5.1.0-beta2 release (#2144) * update release guide steps with new info based on changes to GitHub web interface * update version for development towards rc1 release * Feature #1977 ARGO use case (#2148) Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * Feature develop cycle change (#2150) * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * Added paths to ignore --------- Co-authored-by: jprestop <jpresto@ucar.edu> * Documentation: Update Truth Data instructions (#2154) * upgrade python to 3.10 for documentation builds to fix failured introduced by urllib3 (see urllib3/urllib3#2168) * update workflow to update input test data to use branch from pull down or enter branch by hand if branch does not exist in dtcenter/METplus repo, e.g. if updating data for a PR coming from a fork * Bugfix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) * changed order of workflow event triggering info to more easily see the repository and commit sha that triggered the workflow * fix docker compose install and call -- needed due to new Alpine Linux release 3.18.0 that does not include docker-compose in the apk repository * Bugfix #2168 develop - StatAnalysis time shift (#2169) * Feature 1978 update pyreq doc (#2177) * Updating for 5.1.0 * Adding greater than or equal to signs * Feature 2106 release support (#2182) * Added a paragraph for Release Support Policy * Added a sentence about contacting the team through the METplus GitHub Discussions Forum. * Added Release Support Policy info * Per #2159, modify instructions to reflect that we only want a coordinated release for the official releases and updates to the coordinated release section for the bugfix releases. (#2183) * Feature 2147 workflow doc (#2185) * Add new images for github workflow * Added PR section * Modified the Find the GitHub Issue to be Find or Create a GitHub issue * Trying out image keyword compared with figure keyword * Trying out various widths * Added new figures * Reverting to figures from images after modifying figures * Adding the old images which are better quality * Trying out new indention * Working on formatting again * Reducing image widths * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update github_workflow.rst --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * added scripts to create new conda environment for PANDA-C use cases * Bugfix #2189 develop - spaces in complex thresholds (#2191) * added release notes for completed issues for rc1 * Feature 2011 tci from cesm fluxnet2015 netcdf (#2192) Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Bugfix #2179 develop TCPairs fix -diag argument (#2187) * sort list of files to make command more readable and make logic more easily testable * fix unit test to expect alphabetical order of files * Feature #2197 MODE new multi-variate options (#2203) * added to release notes * added release notes for other items completed for rc1 release * Feature #1626 TCDiag Wrapper (#2201) * Feature #1610 first round of SonarQube fixes (#2200) * Feature #2195 create v5.1.0-rc1 release (#2204) * Update version for development towards official release * updated script to use metplotpy as base and install pygrib because conda was failing trying to install matplotlib in Docker for some strange reason * Feature 2188 use case panda c (#2202) * added first use case, need to finish docs and add to list * added documentation files, 3 new cases * added 5th use case, docs * final use case, corrected paths for output * Update GridStat_fcstGFS_obsERA5_lowAndTotalCloudFrac.py * removed chars, added space to last section * turn off 5/6 new air quality and comp use cases because they are causing the disk to fill up in the automated tests and only 1 is needed to test the new behavior. Reordered use case groups to be alphabetical * disabled other new air quality use case because the disk is still filling up from this case * Feature 2136 cross spectra (#2208) * per #2212, continue instead of return so other forecast leads will not be skipped * Feature #2210 MODE multi-variate variable rename (#2211) * Bugfix diff tests (#2217) * added change from NCEP produtil that was somehow not included in the changes pulled from their repo before starting SonarQube work * Feature #2228 v5.1.0-rc2 release (#2229) * update version after rc2 release to note development towards official 5.1.0 release * Feature #2188 move PANDA-C cases to clouds category and fix automated tests (#2237) Co-authored-by: j-opatz <jopatz@ucar.edu> * Feature 2198 use case amdar pbl (#2206) Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Bugfix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) * Feature #2230 MvMODE usecase doc updates (#2239) This updates the use-case documentation to go along with the multivariate MODE enhancements. * combine release notes from development releases for 5.1.0 release * fix indentation to prevent incorrect nesting of drop down menus * Bugfix #2241 Create directory containing -out_stat file (#2242) * Enhance diff logic to control decimal precision to prevent failures (#2243) * Bugfix #2245 use unique run ID to name logger instance (#2247) * per #2245, add METplusConfig class function that is called when object is deleted to close log handlers. This prevents OSError: [Errno 24] Too many open files from running all pytests * Feature #1626 TCDiag updates to wrapper and basic use case (#2248) Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> * move new use case into another group * Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root <root@localhost> * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <tracy.hertneky@noaa.gov> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: John Sharples <41682323+John-Sharples@users.noreply.github.com> Co-authored-by: root <root@localhost> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: reza-armuei <144857501+reza-armuei@users.noreply.github.com> Co-authored-by: Tracy <tracy.hertneky@noaa.gov> Co-authored-by: Mallory Row <mallory.row@noaa.gov> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com>

@JohnHalleyGotway

* Feature 2065 v5.1.0 beta1 (#2066) * updated version number * added release notes * Update docs/Users_Guide/release-notes.rst Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * updated version with -dev * adding sphinx design to 3 files so dropdown menus will work in the release notes. * fix version numbers for python requirements for building documentation * fixed tab vs spaces * Bugfix #2070 var list numeric order (#2072) * Adding files for jet and hera * added commented process list that was accidentally removed -- this is used if the pre-processing steps are needed * Bugfix #2087 develop docs_pdf (#2091) * Feature 2076 release dropdown menu (#2083) * test dropdowns * Update release-notes.rst Attempting to resolve WARNING * testing dropdowns * adding dropdown menus * testing dropdown remove from warning section * dropdown testing * dropdown menu tests * dropdown menus after the warnings section, the dropdown menu only has one space instead of the normal 2 spaces. If it has 2 spaces, it gets sucked into the warning section. * spaces with warning * no spaces for warning * Update release-notes.rst warning note has to be indented 1 space. Originally it was no spaces but then all of the dropdown menus end up in the warning message. If it's 2 spaces, as is normal, then the warning is indented, which is unnecessary. 1 space is slightly indented and keeps the dropdown menus separate. --------- Co-authored-by: jprestop <jpresto@ucar.edu> * Bugfix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) * Feature 1516 use case mvmode (#2094) Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Fix for Dockerfile smell DL4000 (#2112) Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> * Changed "PROJECT" to "CYCLE" ASSIGNMENT * Feature 2115 update use cases (#2133) * Bugfix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) * Bugfix #2137 develop PointStat -obs_valid_beg/end (#2141) * Feature #2143 5.1.0-beta2 release (#2144) * update release guide steps with new info based on changes to GitHub web interface * update version for development towards rc1 release * Feature #1977 ARGO use case (#2148) Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * Feature develop cycle change (#2150) * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * Added paths to ignore --------- Co-authored-by: jprestop <jpresto@ucar.edu> * Documentation: Update Truth Data instructions (#2154) * upgrade python to 3.10 for documentation builds to fix failured introduced by urllib3 (see urllib3/urllib3#2168) * update workflow to update input test data to use branch from pull down or enter branch by hand if branch does not exist in dtcenter/METplus repo, e.g. if updating data for a PR coming from a fork * Bugfix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) * changed order of workflow event triggering info to more easily see the repository and commit sha that triggered the workflow * fix docker compose install and call -- needed due to new Alpine Linux release 3.18.0 that does not include docker-compose in the apk repository * Bugfix #2168 develop - StatAnalysis time shift (#2169) * Feature 1978 update pyreq doc (#2177) * Updating for 5.1.0 * Adding greater than or equal to signs * Feature 2106 release support (#2182) * Added a paragraph for Release Support Policy * Added a sentence about contacting the team through the METplus GitHub Discussions Forum. * Added Release Support Policy info * Per #2159, modify instructions to reflect that we only want a coordinated release for the official releases and updates to the coordinated release section for the bugfix releases. (#2183) * Feature 2147 workflow doc (#2185) * Add new images for github workflow * Added PR section * Modified the Find the GitHub Issue to be Find or Create a GitHub issue * Trying out image keyword compared with figure keyword * Trying out various widths * Added new figures * Reverting to figures from images after modifying figures * Adding the old images which are better quality * Trying out new indention * Working on formatting again * Reducing image widths * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update github_workflow.rst --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * added scripts to create new conda environment for PANDA-C use cases * Bugfix #2189 develop - spaces in complex thresholds (#2191) * added release notes for completed issues for rc1 * Feature 2011 tci from cesm fluxnet2015 netcdf (#2192) Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Bugfix #2179 develop TCPairs fix -diag argument (#2187) * sort list of files to make command more readable and make logic more easily testable * fix unit test to expect alphabetical order of files * Feature #2197 MODE new multi-variate options (#2203) * added to release notes * added release notes for other items completed for rc1 release * Feature #1626 TCDiag Wrapper (#2201) * Feature #1610 first round of SonarQube fixes (#2200) * Feature #2195 create v5.1.0-rc1 release (#2204) * Update version for development towards official release * updated script to use metplotpy as base and install pygrib because conda was failing trying to install matplotlib in Docker for some strange reason * Feature 2188 use case panda c (#2202) * added first use case, need to finish docs and add to list * added documentation files, 3 new cases * added 5th use case, docs * final use case, corrected paths for output * Update GridStat_fcstGFS_obsERA5_lowAndTotalCloudFrac.py * removed chars, added space to last section * turn off 5/6 new air quality and comp use cases because they are causing the disk to fill up in the automated tests and only 1 is needed to test the new behavior. Reordered use case groups to be alphabetical * disabled other new air quality use case because the disk is still filling up from this case * Feature 2136 cross spectra (#2208) * per #2212, continue instead of return so other forecast leads will not be skipped * Feature #2210 MODE multi-variate variable rename (#2211) * Bugfix diff tests (#2217) * added change from NCEP produtil that was somehow not included in the changes pulled from their repo before starting SonarQube work * Feature #2228 v5.1.0-rc2 release (#2229) * update version after rc2 release to note development towards official 5.1.0 release * Feature #2188 move PANDA-C cases to clouds category and fix automated tests (#2237) Co-authored-by: j-opatz <jopatz@ucar.edu> * Feature 2198 use case amdar pbl (#2206) Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Bugfix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) * Feature #2230 MvMODE usecase doc updates (#2239) This updates the use-case documentation to go along with the multivariate MODE enhancements. * combine release notes from development releases for 5.1.0 release * fix indentation to prevent incorrect nesting of drop down menus * Bugfix #2241 Create directory containing -out_stat file (#2242) * Enhance diff logic to control decimal precision to prevent failures (#2243) * Bugfix #2245 use unique run ID to name logger instance (#2247) * per #2245, add METplusConfig class function that is called when object is deleted to close log handlers. This prevents OSError: [Errno 24] Too many open files from running all pytests * Feature #1626 TCDiag updates to wrapper and basic use case (#2248) Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> * move new use case into another group * Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root <root@localhost> * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <tracy.hertneky@noaa.gov> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y --------- Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: John Sharples <41682323+John-Sharples@users.noreply.github.com> Co-authored-by: root <root@localhost> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: reza-armuei <144857501+reza-armuei@users.noreply.github.com> Co-authored-by: Tracy <tracy.hertneky@noaa.gov> Co-authored-by: Mallory Row <mallory.row@noaa.gov> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com>

@JohnHalleyGotway

* adding sphinx design to 3 files so dropdown menus will work in the release notes. * fix version numbers for python requirements for building documentation * fixed tab vs spaces * Bugfix #2070 var list numeric order (#2072) * Adding files for jet and hera * added commented process list that was accidentally removed -- this is used if the pre-processing steps are needed * Bugfix #2087 develop docs_pdf (#2091) * Feature 2076 release dropdown menu (#2083) * test dropdowns * Update release-notes.rst Attempting to resolve WARNING * testing dropdowns * adding dropdown menus * testing dropdown remove from warning section * dropdown testing * dropdown menu tests * dropdown menus after the warnings section, the dropdown menu only has one space instead of the normal 2 spaces. If it has 2 spaces, it gets sucked into the warning section. * spaces with warning * no spaces for warning * Update release-notes.rst warning note has to be indented 1 space. Originally it was no spaces but then all of the dropdown menus end up in the warning message. If it's 2 spaces, as is normal, then the warning is indented, which is unnecessary. 1 space is slightly indented and keeps the dropdown menus separate. --------- Co-authored-by: jprestop <jpresto@ucar.edu> * Bugfix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) * Feature 1516 use case mvmode (#2094) Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Fix for Dockerfile smell DL4000 (#2112) Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> * Changed "PROJECT" to "CYCLE" ASSIGNMENT * Feature 2115 update use cases (#2133) * Bugfix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) * Bugfix #2137 develop PointStat -obs_valid_beg/end (#2141) * Feature #2143 5.1.0-beta2 release (#2144) * update release guide steps with new info based on changes to GitHub web interface * update version for development towards rc1 release * Feature #1977 ARGO use case (#2148) Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * Feature develop cycle change (#2150) * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * Added paths to ignore --------- Co-authored-by: jprestop <jpresto@ucar.edu> * Documentation: Update Truth Data instructions (#2154) * upgrade python to 3.10 for documentation builds to fix failured introduced by urllib3 (see urllib3/urllib3#2168) * update workflow to update input test data to use branch from pull down or enter branch by hand if branch does not exist in dtcenter/METplus repo, e.g. if updating data for a PR coming from a fork * Bugfix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) * changed order of workflow event triggering info to more easily see the repository and commit sha that triggered the workflow * fix docker compose install and call -- needed due to new Alpine Linux release 3.18.0 that does not include docker-compose in the apk repository * Bugfix #2168 develop - StatAnalysis time shift (#2169) * Feature 1978 update pyreq doc (#2177) * Updating for 5.1.0 * Adding greater than or equal to signs * Feature 2106 release support (#2182) * Added a paragraph for Release Support Policy * Added a sentence about contacting the team through the METplus GitHub Discussions Forum. * Added Release Support Policy info * Per #2159, modify instructions to reflect that we only want a coordinated release for the official releases and updates to the coordinated release section for the bugfix releases. (#2183) * Feature 2147 workflow doc (#2185) * Add new images for github workflow * Added PR section * Modified the Find the GitHub Issue to be Find or Create a GitHub issue * Trying out image keyword compared with figure keyword * Trying out various widths * Added new figures * Reverting to figures from images after modifying figures * Adding the old images which are better quality * Trying out new indention * Working on formatting again * Reducing image widths * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update github_workflow.rst --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * added scripts to create new conda environment for PANDA-C use cases * Bugfix #2189 develop - spaces in complex thresholds (#2191) * added release notes for completed issues for rc1 * Feature 2011 tci from cesm fluxnet2015 netcdf (#2192) Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Bugfix #2179 develop TCPairs fix -diag argument (#2187) * sort list of files to make command more readable and make logic more easily testable * fix unit test to expect alphabetical order of files * Feature #2197 MODE new multi-variate options (#2203) * added to release notes * added release notes for other items completed for rc1 release * Feature #1626 TCDiag Wrapper (#2201) * Feature #1610 first round of SonarQube fixes (#2200) * Feature #2195 create v5.1.0-rc1 release (#2204) * Update version for development towards official release * updated script to use metplotpy as base and install pygrib because conda was failing trying to install matplotlib in Docker for some strange reason * Feature 2188 use case panda c (#2202) * added first use case, need to finish docs and add to list * added documentation files, 3 new cases * added 5th use case, docs * final use case, corrected paths for output * Update GridStat_fcstGFS_obsERA5_lowAndTotalCloudFrac.py * removed chars, added space to last section * turn off 5/6 new air quality and comp use cases because they are causing the disk to fill up in the automated tests and only 1 is needed to test the new behavior. Reordered use case groups to be alphabetical * disabled other new air quality use case because the disk is still filling up from this case * Feature 2136 cross spectra (#2208) * per #2212, continue instead of return so other forecast leads will not be skipped * Feature #2210 MODE multi-variate variable rename (#2211) * Bugfix diff tests (#2217) * added change from NCEP produtil that was somehow not included in the changes pulled from their repo before starting SonarQube work * Feature #2228 v5.1.0-rc2 release (#2229) * update version after rc2 release to note development towards official 5.1.0 release * Feature #2188 move PANDA-C cases to clouds category and fix automated tests (#2237) Co-authored-by: j-opatz <jopatz@ucar.edu> * Feature 2198 use case amdar pbl (#2206) Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Bugfix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) * Feature #2230 MvMODE usecase doc updates (#2239) This updates the use-case documentation to go along with the multivariate MODE enhancements. * combine release notes from development releases for 5.1.0 release * fix indentation to prevent incorrect nesting of drop down menus * Bugfix #2241 Create directory containing -out_stat file (#2242) * Enhance diff logic to control decimal precision to prevent failures (#2243) * Bugfix #2245 use unique run ID to name logger instance (#2247) * per #2245, add METplusConfig class function that is called when object is deleted to close log handlers. This prevents OSError: [Errno 24] Too many open files from running all pytests * Feature #1626 TCDiag updates to wrapper and basic use case (#2248) Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> * move new use case into another group * Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root <root@localhost> * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <tracy.hertneky@noaa.gov> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> --------- Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: John Sharples <41682323+John-Sharples@users.noreply.github.com> Co-authored-by: root <root@localhost> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: reza-armuei <144857501+reza-armuei@users.noreply.github.com> Co-authored-by: Tracy <tracy.hertneky@noaa.gov> Co-authored-by: Mallory Row <mallory.row@noaa.gov> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com>

georgemccabe added 5 commits March 2, 2023 12:45

per #2070, added test to confirm VAR<n> variables are processed in al…

c9f0e02

…phabetical order (VAR1, VAR10, VAR2, etc.) instead of numeric

added comment to describe what test accomplishes

6713125

clean up doc comment

8fde45f

per #2070, sort var list by integer of index

3446691

fix comment

e9658fa

georgemccabe added this to the METplus-5.1.0 milestone Mar 2, 2023

georgemccabe requested a review from JohnHalleyGotway March 2, 2023 20:24

changed index of var (field) list to an integer instead of a string r…

bb261cd

…epresentation of the integer index

georgemccabe mentioned this pull request Mar 8, 2023

Bugfix #2070 main_v5.0 var list numeric order #2074

Merged

JohnHalleyGotway approved these changes Mar 9, 2023

View reviewed changes

georgemccabe merged commit c19aebc into develop Mar 10, 2023

georgemccabe deleted the bugfix_2070_develop_var_order branch March 10, 2023 15:11

georgemccabe linked an issue May 23, 2023 that may be closed by this pull request

Bugfix: Define the order of the forecast variables numerically rather than alphabetically #2070

Closed

23 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bugfix #2070 var list numeric order #2072

Bugfix #2070 var list numeric order #2072

georgemccabe commented Mar 2, 2023

JohnHalleyGotway left a comment •

edited

Loading

Bugfix #2070 var list numeric order #2072

Bugfix #2070 var list numeric order #2072

Conversation

georgemccabe commented Mar 2, 2023

Pull Request Testing

Pull Request Checklist

JohnHalleyGotway left a comment • edited Loading

Choose a reason for hiding this comment

JohnHalleyGotway left a comment •

edited

Loading